Skip to content

[AISAGE-733] perf(gateway): combine provider metadata uploads - #9711

Merged
chrarnoldus merged 5 commits into
mainfrom
combine-ai-provider-metadata-upload
Nov 4, 2026
Merged

chrarnoldus merged 5 commits into
mainfrom
combine-ai-provider-metadata-upload

Conversation

@chrarnoldus

@chrarnoldus chrarnoldus commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Combine the Kilo provider-routing decision and Vercel provider metadata into one post-response R2 object per usage ID. Requests that previously wrote both objects now make 1 PUT instead of 2; routing-only requests stay at 1, and missing archive configuration disables uploads.
  • Harvest full, opaque metadata during the existing raw processUsage parsing pass rather than rewriteModelResponse. The rewriter still strips provider metadata and rewrites client-visible usage. No new stream pass, inference-blocking upload, or database write is introduced.
  • Store <usage_id>.json at the bucket root, containing version-1 routing fields and optional vercel: { metadata }. Preserve the provider's generation ID inside its raw metadata rather than storing a redundant copy.
  • Independent of [AISAGE-733] perf(api-request-log): combine request and response uploads #9709; no api_request_log changes.

Verification

No manual UI, live-provider, or live-R2 testing. This is a server-side archive change; automated tests use local PostgreSQL and an in-memory R2 client.

Automated Checks

  • 12 focused Jest suites passed, 1,442 tests: archive collectors, raw archive/accounting integration, all three usage parsers, usage accounting, response rewriting, LLM proxy/handler helpers, and routing decisions.
  • Final archive and all three API parser rerun with --detectOpenHandles: 275 tests across 6 suites passed and exited cleanly. The broader suite retains its existing open-handle warning from Sentry/rewriter test resources; that runner was stopped after all tests passed.
  • Local verification used an isolated worker database after the shared worker database was reset during the first attempt. No test-harness changes are included in this PR.
  • pnpm --filter @kilocode/web-shared typecheck, pnpm --filter ai-gateway typecheck, and pnpm exec tsgo --noEmit -p apps/web/tsconfig.json passed.
  • Scoped pnpm format, oxfmt --check, oxlint, and git diff --check passed.
  • Merged current origin/main (fe50ac74d4) without rewriting published history.
  • Independent review completed. CI and Kilo automated review passed for final synchronized head 5a72089585; all seven review threads are resolved and no new issues were found.

Visual Changes

N/A

Reviewer Notes

  • The routing callback registers one after() callback without starting I/O. countAndStoreUsage synchronously attaches the Vercel raw-parsing completion promise. Serialization waits for parsing settlement or the first protocol terminal, not generation lookup, pricing, billing publication, or database persistence.
  • Preserve both metadata aliases, all harvested envelopes/choices, unknown provider fields, latest-object replacement semantics, terminal-frame metadata, and observed prefixes after aborts or malformed events. Missing usage or response IDs do not discard observed metadata. Post-terminal frames cannot overwrite it.
  • Pre-accounting rejection/error paths retain routing-only archives. Non-Vercel requests never wait for or harvest Vercel metadata. Archive registration/upload failures remain isolated from responses and billing.
  • Recognized upstream application/json / text/event-stream selects the actual response parser, with requested streaming mode as fallback. JSON fallback usage is accounted rather than discarded, and the stored streamed flag reflects the actual response format. Unterminated-SSE-event behavior is unchanged.
  • Metadata capture in Chat Completions, Messages, and Responses sits beside typed metadata extraction, before error reporting. Event-local names distinguish metadata from accumulated billing state. Forwarding [DONE] is a completion notification, not metadata extraction; the held-open-stream regression proves it releases capture before transport EOF while accounting still drains normally.
  • No database migration, new configuration, web reader, or web-first deployment ordering is required. There are no in-repository consumers of the AI-provider metadata archive. No prefix compatibility or dual-write path is added.

Models Used

  • OpenAI GPT-6.1 Sol for architecture, implementation, integration tests, and independent review.
  • Subagent override: kilo/openai/gpt-6-luna, reasoning high, for archive/consumer research, mechanical rewriter/naming changes, and handler regression coverage.

@chrarnoldus chrarnoldus self-assigned this Nov 3, 2026
@kilo-code-bot

kilo-code-bot Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Code Review Summary

Status: No Issues Found | Recommendation: Merge

Executive Summary

The follow-up commit only relocates and renames event-local provider-metadata extraction in the three usage parsers; the logic is behavior-preserving and no new defects were found.

Assumptions: [INFERENCE] captureVercelMetadata continues to receive the full raw frame/envelope, so moving it below the JSON-null guard and beside the narrower provider_metadata projection does not change what the collector archives.

Files Reviewed (3 files)
  • packages/web-shared/src/lib/ai-gateway/processUsage.messages.ts
  • packages/web-shared/src/lib/ai-gateway/processUsage.responses.ts
  • packages/web-shared/src/lib/ai-gateway/processUsage.ts
Previous Review Summaries (2 snapshots, latest commit 36d6967)

Current summary above is authoritative. Previous snapshots are kept for context only.

Previous review (commit 36d6967)

Status: No Issues Found | Recommendation: Merge

Executive Summary

The follow-up commit removes the redundant generation-ID tracking, flattens the archive key to <usage_id>.json, and consolidates the Messages metadata capture beside the existing extraction; the refactor is behavior-preserving and no new defects were found.

Assumptions: [INFERENCE] The provider's generation ID remains available inside the raw provider_metadata (e.g. gateway.generationId), so dropping the separate generation_id field loses nothing; no in-repository consumer of the old kilo/ key prefix or vercel.generation_id field was found.

Files Reviewed (10 files)
  • ENVIRONMENT.md
  • packages/web-shared/src/lib/ai-gateway/handlers/llm-proxy.test.ts
  • packages/web-shared/src/lib/ai-gateway/processUsage.messages.test.ts
  • packages/web-shared/src/lib/ai-gateway/processUsage.messages.ts
  • packages/web-shared/src/lib/ai-gateway/provider-metadata-archive.test.ts
  • packages/web-shared/src/lib/ai-gateway/providers/routing-decision.ts
  • packages/web-shared/src/lib/r2/kilo-routing-decision.test.ts
  • packages/web-shared/src/lib/r2/kilo-routing-decision.ts
  • packages/web-shared/src/lib/r2/vercel-provider-metadata.test.ts
  • packages/web-shared/src/lib/r2/vercel-provider-metadata.ts

Previous review (commit 1287f8e)

Status: No Issues Found | Recommendation: Merge

Executive Summary

Reviewed the combined provider-metadata archive change across the gateway, usage parsers, and R2 helpers; the new collector semantics, deferral/race behavior, and parser-selection change match the behavior they replace, and no high-confidence defects were found in the changed code.

Assumptions: [INFERENCE] No in-repo reader/export of the retired vercel/<generation_id>.json key was found; external generation-key-only tooling and bucket lifecycle policies were not verifiable from this repository. Upstream content-type headers are assumed to match the response body they label.

Files Reviewed (17 files)
  • ENVIRONMENT.md
  • packages/web-shared/src/lib/ai-gateway/handlers/llm-proxy.test.ts
  • packages/web-shared/src/lib/ai-gateway/handlers/llm-proxy.ts
  • packages/web-shared/src/lib/ai-gateway/processUsage.messages.ts
  • packages/web-shared/src/lib/ai-gateway/processUsage.responses.ts
  • packages/web-shared/src/lib/ai-gateway/processUsage.ts
  • packages/web-shared/src/lib/ai-gateway/processUsage.types.ts
  • packages/web-shared/src/lib/ai-gateway/provider-metadata-archive.test.ts
  • packages/web-shared/src/lib/ai-gateway/providers/routing-decision.ts
  • packages/web-shared/src/lib/ai-gateway/rewriteModelResponse.shared.ts
  • packages/web-shared/src/lib/ai-gateway/rewriteModelResponse.test.ts
  • packages/web-shared/src/lib/ai-gateway/rewriteModelResponse.ts
  • packages/web-shared/src/lib/r2/ai-provider-metadata-bucket.ts
  • packages/web-shared/src/lib/r2/kilo-routing-decision.test.ts
  • packages/web-shared/src/lib/r2/kilo-routing-decision.ts
  • packages/web-shared/src/lib/r2/vercel-provider-metadata.test.ts
  • packages/web-shared/src/lib/r2/vercel-provider-metadata.ts

Reviewed by deepseek-flash-latest · Input: 39.1K · Output: 4.2K · Cached: 444.4K

Review guidance: REVIEW.md from base branch main

Comment thread ENVIRONMENT.md Resolved
@joshburt
joshburt self-requested a review November 3, 2026 22:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants